Document review management system

ABSTRACT

An online review system determines scores for document authors and document reviewers. An author score is based on a number of comments added to a document by reviewers. One way to compute the author score is to use a ratio of a number of comments provided by reviewers of a document to a number of lines of document contents provided by the author. A reviewer score for a reviewer is based on an amount of comments subsequently made by other reviewers, and on the document content itself, such as a ratio of a number of comments provided in subsequent reviews of a document to a number of lines of original document contents. In some embodiments, the online review system monitors trends in scores of individuals or teams.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/867,450, filed May 5, 2020, now U.S. Pat. No. 11,157,504, which iscontinuation of U.S. patent application Ser. No. 16/293,452, filed Mar.5, 2019, now U.S. Pat. No. 10,650,010, which is a continuation of U.S.patent application Ser. No. 15/449,785, filed Mar. 3, 2017, now U.S.Pat. No. 10,261,953, all of which are incorporated by reference herein.

FIELD OF ART

This disclosure relates generally to content review systems, and inparticular to scoring content and content reviews.

BACKGROUND

Written content, such as code for software programs and other writtendocuments, often benefits from peer review. Software development, oneexample of content creation, may be overseen by managers such as projectleads who spend many working hours reviewing code produced by theirdevelopers and appraising peer reviews provided by other developers,such as team members. Complex software projects can involve manydevelopers in many locations, and reviewing code and code reviews insuch a situation can be time-consuming and complicated for intermediatereviewers and project leads. A complicated reviewing process may resultin a loss of productivity and poor content quality due to a possiblelack of valuable peer and manager feedback and due to having no way ofmonitoring changes in quality of content and content reviews over time.

SUMMARY

An online system collects content (e.g., as contained in documents) fromauthors and collects content reviews from participating reviewers.Authors upload their documents to the online system and receivefeedback, reviews, ratings, or other recommendations from reviewers.Some examples of reviewers who may review a document include peerreviewers, managers, people managers, project managers,quality-assurance agents, quality assurance engineers, publicationeditors, technical leads, subject-matter experts, educators, an authorproviding a self-review, artificial intelligence agents, and members ofthe public. In some embodiments, reviewers provide feedback for adocument by adding comments inline in the document. Comments may includesuggestions for improvements, citations of logical errors, and ideas forstylistic changes, for example. In some embodiments, reviewers can alsoinclude positive feedback. In some embodiments, a set of predefinedrules may be used to include metadata indicating why a reviewer includeda comment in a document. Such rules may be associated with weightingvalues, such that a score based on a number of comments received may beinfluenced by the specific comment types.

The online system receives document reviews from one or more reviewers.In one embodiment, the online system manages the reviewing process byrequesting a review from a reviewer when a new review is needed. In someembodiments, a review comprises one or more comments added to a documentby a reviewer. In some embodiments, a review may additionally includezero comments added to a document by a reviewer, as when the reviewerviews the document but decides that the document does not need anycommentary. In some embodiments, the online system determines scores forauthors. Such author scores are based on a relationship between thenumber of comments provided by reviewers and the number of lines ofcontent under review in the document, such as a ratio thereof. Theonline system may also determine scores for reviewers. In someembodiments, a reviewer score is based on a relationship between anumber of comments provided by subsequent reviewers and a number oflines of content under review (e.g., a ratio). In some embodiments,scores account for rules that have been associated with specificcomments by incorporating weights that were assigned to the rules in thescore calculations.

The online system provides the determined author scores and reviewerscores for evaluation by participants, managers, and other relatedparties. In some embodiments, the online system additionally monitorstrends of the scores over time, sorts scores by teams, generates reportsabout user scores, and/or identifies score anomalies.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level block diagram that illustrates an environment ofan online review system, in accordance with an embodiment.

FIG. 2A illustrates an example user interface for an online reviewsystem, in accordance with an embodiment.

FIG. 2B illustrates a second example user interface for an online reviewsystem, in accordance with an embodiment.

FIG. 3 is a high level block diagram that illustrates a systemarchitecture for a metric module, in accordance with an embodiment.

FIG. 4 is a flowchart that illustrates an overall process for scoringcontent and content reviews, in accordance with an embodiment.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION

System Environment

Embodiments described herein provide for collection and analysis ofdocuments and reviews of document contents in an online review system.In one embodiment, the online review system analyzes reviewed documentsto determine scores for authors and reviewers who provided feedbackabout the content. The online review system may further use the scoresto monitor trends related to the quality and efficiency of work done byauthors and reviewers. The systems and methods described herein providea tool for authors, reviewers, and managers to keep track of teamperformance.

FIG. 1 is a high level block diagram that illustrates an environment ofan online review system 100, in accordance with an embodiment. Thesystem environment shown in FIG. 1 includes one or more client devices105, a network 120, and an online review system 100. In someembodiments, a client device 105 may include a development environment103. The online review system comprises a metric module 180, a versioncontrol module 170, and a document repository 160.

A client device 105 is a computing device capable of receiving userinput and transmitting and/or receiving data via the network 120. In oneembodiment, a client device 105 is a conventional computer system, suchas a desktop or a laptop computer. Alternatively, a client device 105may be a device having computer functionality, such as a personaldigital assistant (PDA), a mobile telephone, a smartphone, or anothersuitable device. A client device 105 is configured to communicate viathe network 120. In one embodiment, a client device 105 executes anapplication allowing a user of the client device 105 to interact withthe online review system 100. For example, a client device 105 executesa browser application to enable interaction between the client device105 and the online system 100 via the network 120. In anotherembodiment, a client device 105 interacts with the online system 100through an application programming interface (API) running on a nativeoperating system of the client device 105, such as IOS® or ANDROID™.

The client devices 105 are configured to communicate via the network120, which may comprise any combination of local area and/or wide areanetworks, using both wired and/or wireless communication systems. In oneembodiment, the network 120 uses standard communications technologiesand/or protocols. For example, the network 120 includes communicationlinks using technologies such as Ethernet, 802.11, worldwideinteroperability for microwave access (WiMAX), 3G, 4G, code divisionmultiple access (CDMA), digital subscriber line (DSL), etc. Examples ofnetworking protocols used for communicating via the network 120 includemultiprotocol label switching (MPLS), transmission controlprotocol/Internet protocol (TCP/IP), hypertext transport protocol(HTTP), simple mail transfer protocol (SMTP), and file transfer protocol(FTP). Data exchanged over the network 120 may be represented using anysuitable format, such as hypertext markup language (HTML) or extensiblemarkup language (XML). In some embodiments, all or some of thecommunication links of the network 120 may be encrypted using anysuitable technique or techniques.

In some embodiments, a client device 105 includes a developmentenvironment 103. A development environment 103 is a softwareapplication, e.g., with text editing capabilities, through which a usercan develop content such as code. The development environment 103 mayfacilitate content reviews by including reviewing functionality fordisplaying and adding document reviews from within the environment. Insome embodiments, the development environment 103 may also providefunctionality for transmitting content from client device 105 to onlinereview system 100 (e.g., integrated document uploading functionality).

In some embodiments, the development environment 103 is an integrateddevelopment environment (IDE). An IDE is a software program thatprovides a software developer with a wide variety of tools that aid inthe process of software development. These tools provide functionalityfor one or more of editing, compiling, linking, executing, testing,debugging, and profiling source code.

The online review system 100 receives content (i.e., code, writtendocuments, etc.) from authors who submit the content from a clientdevice 105 over the network 120. The online review system 100facilitates one or more reviews of the content by providing the contentto reviewers. Reviewers may include peers, managers, instructors, orother users who read the content and provide feedback. In oneembodiment, the author may provide feedback on the author's ownsubmitted work (e.g., a self-evaluation). The online review system 100scores authors and reviewers based on subsequent reviews. The onlinereview system uses the scores to provide data about trends forindividuals and teams. For example, the online system may provide areport to a manager, indicating that a team the manager supervises hasreceived many comments about the poor quality of their code submissions,and the manager may subsequently make adjustments to a workflow for theteam when planning an upcoming project. In some embodiments, the onlinereview system 100 stores information about which authors, peers,managers, instructors, or other users are on a team or on teams togetherso it can transmit information about scores and trends to relevantusers.

The document repository 160 stores content from authors received at theonline system. The document repository 160 may also store reviews ofdocuments, submitted by reviewers. In some embodiments, the onlinereview system 100 accesses documents stored elsewhere on the network120, for example on a remote server. In one embodiment, the documentrepository 160 allows multiple versions of documents. In one embodiment,the document repository 160 stores only updated portions of new versionsof previously submitted content.

In one embodiment, the online review system 100 includes a versioncontrol module 170. The version control module 170 determines theportions of an uploaded document that are different from previouslyreceived versions. In some embodiments, the version control module 170notifies one or more reviewers of the document of the portions of thedocument that differ from the prior versions, so that the reviewers canmore efficiently review and comment on new material, and collects anydocument reviews that the reviewers produce.

The metric module 180 determines scores for developers and reviewersbased on comments received from reviewers in regards to documents theyare assigned to review. The metric module 180 is discussed in moredetail in a description of FIG. 3 below.

Comments are written critiques, observations, remarks, and the like,referencing part or all of a document. In some embodiments, comments areincluded inline in a document. Thus, a line of original document contentmay also be a line where a comment has been added by a reviewer.Comments may also be included on separate document lines. It may also bepossible for one comment to span multiple lines of a document, as when acomment has been formatted to include newline characters (e.g., to aidreadability). In some embodiments, comments may be indicated byinclusion of one or more predefined symbols. For example, in someprogramming languages, inline comments may be delineated by a doubleforward slash symbol “//” or a pound sign “#” that is used to partitiona line into original document contents and a comment portion. As anotherexample, some programming languages include symbols that represent blockcomments (i.e., individual comments that may span multiple lines), suchas by beginning a comment with “/*” and ending a comment with “*/” Insome embodiments, comments may be indicated using metadata separate fromthe content itself, rather than being inline, such as instances ofpredefined metadata flags indicating various situations (such as a flagindicating that associated code should be checked for errors, that thestyle of the associated code does not conform to the proper standard, orthe like).

User Interfaces

FIG. 2A illustrates an example user interface for an online reviewsystem, in accordance with an embodiment. In some embodiments, theonline review system 100 determines scores and trends for participatingauthors and reviewers and presents the scores and trends via a userinterface for monitoring by the authors, reviewers, and theirsupervisors or instructors. For example, a web page displaying trendsand scores may be accessed via a web browser. Some embodiments of theonline review system 100 do not include an interactive user interface.For example, an online review system 100 can alternatively provide scorevalues and trend predictions in a raw data format (e.g., stored in plaintext files) for an author, reviewer, or manager to assess.

The example of FIG. 2A depicts a user interface 210 through which aproject lead may track a software development team. The user interface210 may be presented as a web page via a web browser, or may bepresented by another software application. In the embodiment shown inFIG. 2A, trends for individual software developers on a development teamfor a project are presented for assessment by a manager or project lead.A trend graph 215 is displayed for each developer on the team. The trendgraph 215 represents the changes in scores assigned to the developer bythe online review system 100 over time, for example, over the course ofseveral review cycles for a project. A trend graph may include an authorscore trend 220 and a reviewer score trend 225 for the developer.

An author score trend 220 tracks the scores an author (i.e., developer,writer, composer, etc.) receives for documents the author writes. Areviewer score trend 225 tracks the scores a reviewer receives forreviewing documents submitted by participants. A participant may be bothan author and a reviewer.

A user interface 210 may include listings 230 of content a participantis involved with either as an author or as a reviewer. In someembodiments, the user interface 210 includes an overview pane thatallows reviewers and/or supervisors to view a report that includes dataabout author and reviewer scores for team members associated with aproject. Note that team members may include any individuals whoparticipate in the authorial process or review process and/or have beengranted access to view the data about author and reviewer scores. Insome cases, the user interface 210 includes functionality for reviewersto add comments and feedback to submitted documents via the userinterface. For example, a user may be able to access content to providecomments using software interface elements 235 that open a document forreview and may use additional software elements 240 to display furtherdocuments associated with a participant.

FIG. 2B illustrates another example user interface for an onlinereviewing system, in accordance with an embodiment. In the example ofFIG. 2B, a user interface 210 displays a graph with overlays of authorscore trend lines of participants. Such a graph makes it easy to comparescores and/or improvements made by multiple participants, such asdetermining that Developer 1, though initially receiving lower authorscores than Developer 2, has improved over time and now receives authorscores more favorable than those of Developer 2. An overlay graph mayinclude trend lines 250 and trend identifiers 260 that indicate whichtrend line 250 pertains to a particular participant. The interface 210may include interactive software elements 255 that allow a user toswitch between various trend comparisons. For example, a user may beable to switch between viewing a graph of trend lines 250 related toauthor scores and viewing a graph of trend lines 250 related to reviewerscores, such as by selecting tabs 255.

Metric Module

FIG. 3 is a high level block diagram that illustrates a systemarchitecture for a metric module 180, in accordance with an embodiment.The metric module 180 determines scores to assign to authors andreviewers. Scores are based on the content of documents (e.g., as storedin the document repository 160), such as a number of lines of originaldocument contents and a number of comments provided by reviewers. Themetric module includes a rule repository 310, a metric computationmodule 320, a score repository 330, and a trend development module 340.

The rule repository 310 stores rules that can help reviewers to providecategorized feedback when commenting on a document. A rule is a highlevel policy that defines quality. The use of a rule in conjunction witha comment indicates that the comment relates to a concept or policy thatis defined by the rule. In some embodiments, multiple rules may beassigned to one comment. Assigning a rule to a comment is also hereinreferred to as tagging a comment. Tagging a comment may compriseincluding keywords related to specific rules in comment text, includingspecial characters associated with a rule in a comment, including a linkto metadata associated with a rule in the comment, and so forth. In someembodiments, a reviewer is not required to assign a rule to each commentprovided as review. In some embodiments, no rules are defined forreviewers to associate with document comments. A participant, such as amanager or instructor, may specify a set of rules, for example, to applyto an upcoming code review for a development team. Rules may besubsequently added or edited. When a reviewer provides a review of adocument (e.g., as inline comments), the reviewer can specify one ormore rules that the reviewer considers relevant to the particularsituation or that is the reviewer's reason for including the feedback.For example, a manager may define a “Don't Repeat Yourself” rule thatreviewers can associate with comments in which they point out that theauthor has repeated a logical process that ought to be written as asingle separate function.

In one embodiment, rules stored in the rule repository 310 includeweighting values that indicate how a comment tagged with such a ruleshould be accounted for in a scoring process. As one example, a managermay assign smaller weighting values to rules associated with minorerrors (e.g., stylistic inconsistencies) and may assign larger weightingvalues to rules associated with major errors (e.g., logical errors). Asanother example, a manager wishing to reward documents and reviews thatreceive positive feedback may include a “Positive Feedback” rule thatincludes a negative weighting value, or a weighting value of zero, sothat positive comments either boost a resulting score (lower scoresbeing indicative of superior performance) or are not counted against afinal score of the author or reviewers. In cases where multiple rulesmay be assigned to a comment, the scoring process may account for themultiple rules in a variety of ways. For example, a weighting value ofeach rule may be used to calculate a score, the weight of only a firsttagged rule may be used to calculate a score, a largest or smallest ruleweight may be used to calculate a score, etc.

The metric computation module 320 calculates scores for authors andreviewers of documents stored in the document repository 160. Anauthor's score in relation to a document is based on some or all reviewsprovided for the document (i.e., as comments written by reviewers). Areviewer's score in relation to a document is based on some or allreviews provided for the document after the reviewer added feedback tothe document. That is, a reviewer's score is based on subsequent reviewsto the document.

An author score is calculated for some or all authors who transmitdocuments to the online review system 100. The author score is based ona comparison of the amount of commenting (e.g., number of comments) withthe amount of document content (e.g., number of lines of originaldocument content). In one embodiment, for example, the metriccomputation module counts a number of comments that have been includedin a document by reviewers and compares the amount to a number of linesof original document content submitted by the author. That is, a metricfor scoring an author with respect to a document is the number ofcomments on a document divided by the number of lines of originalcontent (i.e., not reviewer comments) in the document. In embodimentsthat include functionality for tagging comments according to rules withassociated weighting values, the metric for scoring an author withrespect to a document is the sum of the products of the number of eachtype of comment (i.e., as categorized by rules) and the weighting valueassociated with that type of comment, divided by the number of lines oforiginal document content submitted by the author. In some embodiments,a different unit value may be used to reference the number of lines ofcontent in a document; for example, the scoring metric mightalternatively be the number of comments received on a document per every100 lines of content in the document.

A reviewer score is calculated for some or all reviewers who reviewdocuments that have been submitted to the online review system 100 bycommenting on the document contents. A reviewer score is based oncomments added to the document by other reviewers after the addition ofthe comments of the reviewer being scored. In one embodiment, the metriccomputation module counts a number of comments that have been includedin a document by subsequent reviewers and compares it to a number oflines of original document content submitted by the author. That is, ametric for scoring a reviewer with respect to a document is the numberof comments received on a document subsequent to the reviewer's reviewdivided by the number of lines of content in the original document. Forexample, a document might have 25 lines of original content created byan author, 8 of comments provided by a first reviewer, and 4 commentsprovided by a second reviewer. A resulting reviewer score for the firstreviewer would be 4 divided by 25, or 0.16.

In some embodiments that include functionality for tagging commentsaccording to rules with associated weighting values, the metric forscoring a reviewer with respect to a document is the sum of the productsof the number of each type of subsequent comment (i.e., as categorizedby rules) and the weighting value associated with that type of comment,divided by the number of lines of original document content submitted bythe author. As with schemes for scoring authors, in some embodiments,the weights associated with scores may be positive or negative. Inaddition to basing weights on rule type, weights may also be based onother indicators, such as the role or importance of the reviewer, thereviewer scores of other reviewers, and the role or importance of otherreviewers. For example, a subsequent review by a reviewer in amanagerial role might be weighted more heavily than a subsequent reviewby a peer reviewer.

In most situations, each subsequent reviewer for a document will providefewer comments than previous reviewers, given that in most cases theearliest reviewers will catch the most egregious errors (e.g., logicalerrors in code, or typos in a written document). Thus, an implication ofa subsequent reviewer including many comments may be that an earlierreviewer did not do a thorough job at reviewing the document.

Managers, project leads, and instructors may also provide feedback ondocuments. In some cases, reviewers, such as managers, project leads,instructors, or experts, may provide cross-project feedback (i.e.,reviewing a document from a project they are not directly affiliatedwith). In some embodiments, a manager not affiliated with a project mayreceive a reviewer score based on a comparison of the number of commentsthe unaffiliated manager provides on a project document and the numberof comments an affiliated manager provides on the project document.

In some embodiments, a reviewer score previously assigned to a reviewermay influence the extent to which the reviewer's comments areincorporated into reviewer or author scores, e.g., by being applied as aweight for the rating of the reviewer. For example, if a reviewer hasreceived low reviewer scores in the past, the reviewer's comments on adocument may not influence author scores or reviewer scores as much ascomments from a different, more highly rated reviewer.

In some embodiments, the metric module 180 may include an additionalmodule that can automatically evaluate content appropriateness andrelevance. For example the metric module 180 may use preprogrammed rulesand/or trained machine models to analyze comments and detect whether thecomments include content that may be considered sexist, racist, ageist,unrelated to the document contents (e.g., jokes, comments about theweather), and so forth. In such cases, the metric module 180 may assigna weighting factor to a comment or ignore the comment so that it is notincluded in calculated author and reviewer scores. In some embodiments,the metric module 180 may flag such comments for review by a manager(e.g., by adding specific symbols, or by associating the comment with ametadata tag).

According to one evaluation scheme in which no weighting rules are used,a zero represents a perfect score (e.g., there was no reason for anyreviewers to provide feedback or suggestions for improving the document)and scores nearer to one indicate a document or a document review thatreceived a relatively large amount of feedback, and which is thereforelikely in need of improvement. Other evaluation schemes, such as thosethat allow for negative or positive weights (e.g., correspondingrespectively to positive feedback and critical errors), may also beused, and may have a variety of highest and lowest possible scores.

The score repository 330 stores author scores and reviewer scorescomputed by the metric computation module 320. The score repository 330may store scores in association with information about the documentsand/or projects they relate to. Historical scores in the scorerepository 330 may be used to compute trends for individual users orteams. Trends may be computed over individual projects or over multipleprojects.

The trend development module 340 compiles score information from thescore repository 330 to track changes in scores received by authors andreviewers over time. A trend is a dynamic indicator of an author's codequality and of reviews provided by peers. Trend data may be displayed asa graph in a user interface, as depicted in FIG. 2A and FIG. 2B.

Process Flow

FIG. 4 is a flowchart depicting a process for scoring authors andreviewers, in accordance with an embodiment. The online review system100 receives 410 a document from a participating author. A document maybe a written work, such as an essay or a coded software program. Thedocument is stored in a document repository 160. In some embodiments, aversion control module 170 determines whether the document has beenpreviously stored by the document repository 160 and, if so, whatportions of the document have been changed.

The document is transmitted to a reviewer for review (e.g., via email).The reviewer need not be an initial reviewer of the document, which insome cases may have received reviews from other reviewers prior to thereview by the reviewer described here. In some embodiments, only aportion of the document that includes changed content (i.e., as detectedby the version control module 170) is sent to the reviewer for review.The reviewer adds comments to the document. In cases in which thereviewer is not an initial reviewer of the document, the reviewer mayprovide comments about portions of the document that have not beencommented on by previous reviewers and may add additional commentary toclarify or rebut comments from previous reviewers. In some embodiments,the reviewer includes information indicating that particular commentsapply to certain rules, such as may have been predefined by a manager orinstructor. For example, the reviewer may add a metadata tag to eachcomment, associating the comment with a specific rule. The online reviewsystem receives 420 a review of the document from the reviewer. In oneembodiment, receiving 420 the review comprises receiving the entiredocument, with the review itself being included inline (e.g., ascomments) in the document.

In some embodiments, the document and accompanying comments from thereviewer are transmitted to a set of one or more subsequent reviewersfor additional review. In some embodiments, the document transmitted tothe set of subsequent reviewers may only be the changed portion of thedocument, such as the same portion that was provided to the reviewer.The reviewers in the set of subsequent reviewers add comments to thedocument, and may include information indicating that particularcomments apply to certain rules, as was done by the reviewer. In someembodiments, the reviewers in the set of subsequent reviewers receiveand review the document in sequence (i.e., rather than all at once) suchthat each reviewer reviews a document that includes comments fromadditional previous reviewers. The online review system 100 receives 430reviews of the document from the subsequent reviewers.

The online review system 100 determines 440 an author score based on theoriginal document contents and the comments provided by one or morereviewers of the document (e.g., comments provided by the reviewer andcomments provided by the set of subsequent reviewers). In oneembodiment, the metric module 180 computes an author score bydetermining a ratio between the number of comments that reviewers addedto the document, and the number of lines of content in the originaldocument. In some embodiments, rules associated with comments provideweight values indicating to what extent certain comments should beincluded in a count of the number of comments provided for a document.

The online system determines 450 a reviewer score for the reviewer basedon the original document contents, and the comments provided by thereviewers in the set of subsequent reviewers. In some embodiments, themetric module 180 computes a reviewer score by determining a ratiobetween the number of lines of comments that reviewers in the set ofsubsequent reviewers added to the document, and the number of lines inthe original document. In some embodiments, rules associated withcomments provide weight values indicating to what extent certain linesof comments should be included in a count of the number of lines ofcomments provided for a document. Subsequent reviewers (who are not thefinal reviewer) may also receive a reviewer score (e.g., based onreviews added to the document after a subsequent reviewer submits areview). In some embodiments, scores for an author and some or allreviewers may be calculated after a final reviewer (such as a manager)completes a review of a document. In some embodiments, scores forauthors and/or reviewers are calculated (or recalculated) atpredetermined intervals, or each time additional comments are added tothe document.

The online review system 100 provides 460 the determined author scoreand the determined reviewer score (as well as calculated scores of anyother reviewers) for evaluation by the author and reviewers. In someembodiments, the online review system 100 provides the determined scoresfor evaluation by people other than the author or reviewers. Forexample, the online review system 100 may provide the scores tomanagers, project leads, or instructors. In some embodiments, a trenddevelopment module 340 compiles scores of individuals or project teamsto display trends in the scores. For example, a manager may want tomonitor downward, stable, or upward trends in scores to make moreinformed decisions regarding project timelines, project milestones,overall content quality, and expected delays. The online review system100 may generate reports about user scores and/or identify scoreanomalies. For example, the online system 100 may compare an author'smost current score with the author's previous scores and send a messageto a supervisor of the author if the author score changed by more than apredetermined value from past scores received by the author.

Additional Considerations

The present invention has been described in particular detail withrespect to several possible embodiments. Those of skill in the art willappreciate that the invention may be practiced in other embodiments. Theparticular naming of the components, capitalization of terms, theattributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the invention or its features may have different names,formats, or protocols. Further, the system may be implemented via acombination of hardware and software, as described, or entirely inhardware elements. Also, the particular division of functionalitybetween the various system components described herein is merelyexemplary, and not mandatory; functions performed by a single systemcomponent may instead be performed by multiple components, and functionsperformed by multiple components may instead performed by a singlecomponent.

Some portions of above description present the features of the presentinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. These operations, while describedfunctionally or logically, are understood to be implemented by computerprograms. Furthermore, it has also proven convenient at times to referto these arrangements of operations as modules or by functional names,without loss of generality.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “determining” or the like, refer tothe action and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

Certain aspects of the present invention include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present inventioncould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer and run bya computer processor. Such a computer program may be stored in acomputer readable storage medium, such as, but is not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs,magnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, applicationspecific integrated circuits (ASICs), or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus. Furthermore, the computers referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

In addition, the present invention is not limited to any particularprogramming language. It is appreciated that a variety of programminglanguages may be used to implement the teachings of the presentinvention as described herein, and any references to specific languages,such as HTML or HTML5, are provided for enablement and best mode of thepresent invention.

The present invention is well suited to a wide variety of computernetwork systems over numerous topologies. Within this field, theconfiguration and management of large networks comprise storage devicesand computers that are communicatively coupled to dissimilar computersand storage devices over a network, such as the Internet.

Finally, it should be noted that the language used in the specificationhas been principally selected for readability and instructionalpurposes, and may not have been selected to delineate or circumscribethe inventive subject matter. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting, of the scopeof the invention.

What is claimed is:
 1. A computer-implemented method comprising: at anonline system: receiving a document created by an author; receiving afirst review of the document from a first reviewer, the first reviewincluding one or more first comments provided by the first reviewer;receiving one or more second reviews from a set of second reviewers, thesecond reviews including one or more second comments provided byreviewers in the set of second reviewers; storing the document, thefirst review, and the one or more second reviews in a documentrepository associated with the online system; determining a reviewerscore for the first reviewer based on the document and the one or moresecond reviews; determining an author score for the author based on thedocument, the first review, and the one or more second reviews, thedetermining the author score including: calculating a weighted sum of anumber of first comments in the first review and second comments in theone or more second reviews, and computing the author score based on aratio between the weighted sum and a number of lines of the document;storing the reviewer score and the author score in a score repositoryassociated with the online system.
 2. The computer-implemented method ofclaim 1, wherein determining a reviewer score for the first reviewerincludes: computing the reviewer score for the first reviewer based on acomparison of the number of lines of the document and a number of secondcomments in the one or more second reviews.
 3. The computer-implementedmethod of claim 1, wherein at least a subset of the one or more firstcomments and the one or more second comments is tagged with rules, arule from the rules indicating a categorization of a comment of the atleast the subset with which the rule is associated and associating aweighting value with the comment of the at least the subset with whichthe rule is associated.
 4. The computer-implemented method of claim 3,the weighted sum including a sum of products, a product of the productsincluding a count of comments of the at least the subset associated witha rule from the rules and a weighting value associated with the rulefrom the rules.
 5. The computer-implemented method of claim 4, whereindetermining the reviewer score for the first reviewer includes:calculating a weighted sum of a number of second comments added to thedocument by the reviewers in the set of second reviewers, the weightedsum including a sum of products, a product of the products including anumber of comments of the one or more second comments associated with arule from the rules and a weighting value associated with the rule fromthe rules; and computing the reviewer score based on a ratio between theweighted sum and the number of lines of the document.
 6. Thecomputer-implemented method of claim 1, further comprising monitoringtrends in author scores and reviewer scores over time, and providinginformation concerning the trends to team members of the author and teammembers of the first reviewer.
 7. A computing system comprising: one ormore processors; and memory coupled to the one or more processors, thememory storing one or more programs configured to be executed by the oneor more processors, the one or more programs including instructions for:receiving a document created by an author; receiving a first review ofthe document from a first reviewer, the first review including one ormore first comments provided by the first reviewer; receiving one ormore second reviews from a set of second reviewers, the second reviewsincluding one or more second comments provided by reviewers in the setof second reviewers; storing the document, the first review, and the oneor more second reviews in a document repository associated with thecomputing system; determining a reviewer score for the first reviewerbased on the document and the one or more second reviews; determining anauthor score for the author based on the document, the first review, andthe one or more second reviews, the determining the author scoreincluding: calculating a weighted sum of a number of first comments inthe first review and second comments in the one or more second reviews,and computing the author score based on a ratio between the weighted sumand a number of lines of the document; storing the reviewer score andthe author score in a score repository associated with the computingsystem.
 8. The computing system of claim 7, wherein determining thereviewer score for the first reviewer includes computing the reviewerscore for the first reviewer based on a comparison of the number oflines of the document and a number of second comments in the one or moresecond reviews.
 9. The computing system of claim 7, wherein at least asubset of the one or more first comments and the one or more secondcomments is tagged with rules, a rule from the rules indicating acategorization of a comment of the at least the subset with which therule is associated and associating a weighting value with the comment ofthe at least the subset with which the rule is associated.
 10. Thecomputing system of claim 9, the weighted sum including a sum ofproducts, a product of the products including a count of comments of theat least the subset associated with a rule from the rules and theweighting value associated with the rule from the rules.
 11. Thecomputing system of claim 10, wherein determining the reviewer score forthe first reviewer includes: calculating a weighted sum of a number ofsecond comments added to the document by the reviewers in the set ofsecond reviewers, the weighted sum including a sum of products, aproduct of the products including a number of comments of the one ormore second comments associated with a rule from the rules and theweighting value associated with the rule from the rules; and computingthe reviewer score based on a ratio between the weighted sum and thenumber of lines of the document.
 12. The computing system of claim 7,wherein the one or more programs further include instructions formonitoring trends in author scores and reviewer scores over time, andproviding information concerning the trends to team members of theauthor and team members of the first reviewer.